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Abstract. There is growing interest in the use of a multitiered system of supports 
framework to address issues related to school climate and bullying. Positive 
Behavioral Interventions and Supports (PBIS) is one such model that has received 
considerable attention; however, nearly all of the extant literature has focused on 
elementary and middle schools, with limited research on high schools. Further¬ 
more, research on PBIS implementation in high schools, particularly in relation to 
school context, is scant. The current article examined the adoption and imple¬ 
mentation of PBIS in 31 high schools randomly assigned to implement PBIS, 
within the context of a larger 58 high school randomized trial. We first present 
descriptive data on the rollout of the core features of PBIS, as measured by a set 
of research-based implementation tools administered by outside observers. We 
then explore the extent to which baseline rates of bullying and other school-level 
indicators of disorder were associated with the adoption of the multitiered PBIS 
framework over the course of 2 years. Multilevel analyses on the longitudinal 
implementation data indicated that schools with higher baseline rates of bullying 
generally implemented PBIS with greater fidelity over time. This suggests that 
schools with increased bullying may be particularly motivated to adopt PBIS. 
However, other baseline indicators of disorder were generally not associated with 
PBIS implementation and thus do not appear to be barriers to adoption. Impli¬ 
cations for implementation research and practice in high schools are discussed. 


Many high schools struggle to address 
issues related to school climate and bullying 
and thus are turning to school-wide applica¬ 


tions of a multitiered system of supports 
(MTSS) as a framework for addressing these 
concerns. Positive Behavioral Interventions 
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and Supports (PBIS; Sugai & Horner, 2006) is 
one such MTSS framework that has received 
considerable attention. Previous randomized 
trials and rigorous studies of PBIS at the 
elementary school level have demonstrated 
significant impacts on bullying, school cli¬ 
mate, and other disciplinary problems, as 
well as academic performance (see Brad¬ 
shaw, 2013; Bradshaw, Koth, Thornton, & 
Leaf, 2009; Horner et ah, 2009; Horner, 
Sugai, & Anderson, 2010; Waasdorp, Brad¬ 
shaw, & Leaf, 2012). However, there has 
been less investigation of PBIS implemen¬ 
tation or outcomes at the high school level 
(Flannery, Sugai, & Anderson, 2009; for 
notable exceptions of research on PBIS in 
high schools, see Caldarella, Shatzer, Gray, 
Young, & Young, 2011, and Freeman et al., 
2016). 

The current article aimed to extend our 
understanding of the implementation of PBIS 
in high schools, with particular interest in the 
extent to which baseline levels of bullying 
and other indicators of school disorder were 
associated with PBIS implementation over 
the course of 2 years. Our interest in school 
contextual correlates of implementation was 
motivated by prior implementation research, 
largely at the elementary school level 
(Domitrovich et al., 2008; Pas & Bradshaw, 
2012), which suggests that these factors may 
predict adoption and implementation fidelity 
of the three-tiered PBIS model. Further¬ 
more, there is growing awareness of the 
importance of considering fidelity of pro¬ 
gram implementation within the bullying lit¬ 
erature (for a review, see Polanin & Espel- 
age, 2014). These implementation issues 
were explored within the context of 31 high 
schools randomly assigned to the PBIS in¬ 
tervention condition, as a part of a larger 58 
high school randomized controlled trial. 
These findings have particular relevance to 
educators and school psychologists, given 
the widespread dissemination of PBIS and 
scale-up of an MTSS, as well as increasing 
concerns related to bullying in U.S. schools 
(Bradshaw, 2015). 


POSITIVE BEHAVIORAL 
INTERVENTIONS AND SUPPORTS 

PBIS promotes setting-level change as a 
means for systematically and consistently pre¬ 
venting student behavior problems and pro¬ 
moting a positive school environment. The 
model draws on behavioral, social learning, 
organizational, and positive youth develop¬ 
ment theories and promotes strategies that are 
used by all staff consistently across all school 
contexts (Lewis & Sugai, 1999; Lindsley, 
1992; Sugai, Horner, & Gresham, 2002). 
Through PBIS, staff and students work to¬ 
gether to create a school-wide program that 
clearly articulates positive behavioral expecta¬ 
tions, provides incentives to students meeting 
these expectations, promotes positive student- 
staff interactions, and encourages data-based 
decision-making by staff and administrators. 
The model aims to alter the school environ¬ 
ment by creating improved systems (e.g., dis¬ 
cipline, reinforcement, data management) and 
procedures (e.g., collection of office referral 
data, training) and using data-based decision¬ 
making in order to promote positive change in 
student and teacher behaviors (Knoff, 2000; 
Kutash, Duchnowski, & Lynn, 2006; Sugai & 
Horner, 2006; Sugai et al., 2002). The PBIS 
model follows a multitiered prevention approach 
(Mrazek & Haggerty, 1994; O’Connell, Boat, & 
Warner, 2009), whereby Tier 2 (selective) and 
Tier 3 (indicated) programs and supports are 
implemented to complement the Tier 1 (univer¬ 
sal) components (Sugai & Horner, 2006; Walker 
et al., 1996). 

The Tier 1 school-wide PBIS compo¬ 
nent is composed of the following seven crit¬ 
ical features: (a) Within the school, a PBIS 
team is formed that includes 6-10 staff mem¬ 
bers and an administrator, all of whom provide 
building-level leadership regarding the imple¬ 
mentation of PBIS. The team attends annual 
training events, establishes an action plan for 
implementation, develops materials to support 
program implementation, trains other staff 
members, and meets approximately twice a 
month to discuss school-wide behavior man¬ 
agement systems and procedures, (b) A behav¬ 
ioral support coach provides on-site consulta- 
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tion and technical assistance regarding the im¬ 
plementation of PBIS. The coach is typically a 
school psychologist or guidance counselor 
who has prior experience working with PBIS 
and conducting functional behavioral assess¬ 
ments. (c) Expectations for positive student 
behavior are defined and known by staff and 
students. The school team establishes three to 
five positively stated school-wide expectations 
for student behavior (e.g., “Be respectful,” 
“Be responsible,” and “Be ready to learn”), 
which are posted in all school settings, (d) 
Defined behavioral expectations are taught to 
all students. Plans are developed and imple¬ 
mented by the school staff to teach students 
the behavioral expectations, (e) A school-wide 
system is developed to reward students who 
exhibit the expected positive behaviors. 
School staff establish and use a school-wide 
system for reinforcement that includes a tan¬ 
gible reinforcer (e.g., high-fives or bucks) that 
is used consistently by all staff, (f) An agreed- 
upon system is created to respond to behav¬ 
ioral violations. Staff and administrators agree 
on what constitutes classroom- versus office- 
managed discipline problems, and students 
across all classrooms receive consistent con¬ 
sequences for disciplinary infractions, (g) A 
formal system is developed to collect, ana¬ 
lyze, and use data for data-based decision¬ 
making (for additional details, see Horner et 
al., 2010; Sugai & Horner, 2006; Walker et 
al., 1996). 

Although the three-tiered PBIS frame¬ 
work has been widely promoted by the U.S. 
Department of Education and many state de¬ 
partments of education, dissemination and re¬ 
search efforts have largely focused on the 
Tier 1 elements. Consistent with a response- 
to-intervention approach (Hawken, Vincent, & 
Schumann, 2008), it is hypothesized that 80% 
of students will respond to the Tier 1 PBIS 
model and that the remaining 20% of youth 
will need Tier 2 or Tier 3 prevention program¬ 
ming to be successful at school. These selec¬ 
tive and indicated programs are offered to a 
subset of students usually by someone other 
than their classroom teacher (e.g., school psy¬ 
chologist). Moreover, PBIS can serve as a 
framework for the integrated implementation 


of other evidence-based programs (EBPs; 
Bradshaw, Bottiani, Osher, & Sugai, 2014; 
Domitrovich et al., 2010). 

The selection of other EBPs is intended 
to be guided by the collection, review, and use 
of various sources of data. In fact, PBIS 
schools are encouraged to collect multiple data 
elements and receive training on the collection 
of these elements, which are regularly ana¬ 
lyzed and summarized; these data are used by 
the PBIS team to make decisions regarding 
program implementation, as well as to identify 
whether the majority of students are respon¬ 
sive to the universal Tier 1 supports and to 
select other EBPs to meet the needs of stu¬ 
dents. To date, most schools have focused on 
suspensions and office disciplinary referral 
data (Irvin et al., 2006); however, there is 
increasing interest in the use of school climate 
and bullying data as well (Bradshaw, 2013). 
For example, review of school climate data 
may suggest high rates of bullying, and there¬ 
fore, the school may opt to integrate an addi¬ 
tional bullying-focused EBP within the PBIS 
framework to address this particular challenge 
(Bradshaw, 2013). Yet, research suggests that 
most schools struggle to collect and effec¬ 
tively use different types of data to determine 
the most appropriate Tier 2 and 3 interventions 
to meet the needs of nonresponders to the 
Tier 1 supports (Barrett, Bradshaw, & Lewis- 
Palmer, 2008; Debnam, Pas, & Bradshaw, 
2012; Bradshaw, Debnam, et al., 2014). 

Evidence Base for PBIS 

As noted earlier, although PBIS is in¬ 
tended to be multitiered in its implementation, 
most of the wide-scale adoption of the model, 
as well as the research demonstrating its ef¬ 
fectiveness, has focused on Tier 1 supports 
and has been conducted in elementary schools 
(Horner et al., 2010). For example, nonran- 
domized evaluations indicate that training 
schools in PBIS was associated with changes 
in the internal discipline practices (Nersesian, 
Todd, Lehmann, & Watson, 2000); these ef¬ 
fects have been found to persist over several 
years (e.g., Taylor-Greene & Kartub, 2000). 
Schools achieving a high level of PBIS imple- 
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mentation fidelity experienced 20%-60% re¬ 
ductions in office discipline referrals (Kartub, 
Taylor-Greene, March, & Horner, 2000; 
Lewis, Colvin, & Sugai, 2000; Taylor-Greene 
et al., 1997). More recently, there have been 
two randomized controlled trials of Tier 1 
PBIS in elementary schools, which demon¬ 
strated significant impacts on suspensions and 
office referrals, bullying, and peer rejection, as 
well as improved academic achievement and 
school climate (see Bradshaw, Koth, Bevans, 
Ialongo, & Leaf, 2008; Bradshaw, Koth, et al., 
2009; Bradshaw, Mitchell, & Leaf, 2010; 
Bradshaw, Waasdorp, & Leaf, 2012; Horner et 
al., 2009; Waasdorp et al., 2012). Another 
randomized trial combining Tier 1 and 2 sup¬ 
ports (relative to Tier 1 only) in elementary 
schools also demonstrated significant impacts 
on teacher and student behaviors (e.g., special 
education use, need for advanced-tier sup¬ 
ports, teacher efficacy; Bradshaw, Pas, 
Goldweber, Rosenberg, & Leaf, 2012). Al¬ 
though these findings from elementary 
schools are promising, little is known about 
the implementation of PBIS or impacts at 
the secondary level, particularly in high 
schools (Flannery et al., 2009). In fact, some 
practitioners and researchers have suggested 
that implementation of PBIS, particularly 
within the context of integrating other EBPs 
across the advanced tiers, may be more chal¬ 
lenging in high schools and that achieving 
high-fidelity implementation may take lon¬ 
ger (Bradshaw, Debnam, et al., 2014). This 
may be due in part to some organizatio¬ 
nal challenges that high schools face 
(Domitrovich et al., 2008) or misperceptions 
that PBIS components (e.g., need to teach 
and reward behavioral expectations) are not 
developmentally appropriate for high school 
students (Flannery et al., 2009). 

POTENTIAL FACILITATORS AND 
BARRIERS TO PBIS 
IMPLEMENTATION 

Understanding the implementation of 
school-based programs has received increased 
attention in both the conceptual and empirical 
literature (Bruhn, Hirsch, & Lloyd, 2015; For¬ 


man et al., 2013); however, this literature has 
largely focused on implementation in elemen¬ 
tary schools. Several models and frameworks 
have been developed to characterize various 
factors that influence the implementation pro¬ 
cess and program fidelity, such as leadership, 
buy-in, and characteristics of the implement¬ 
ed (Han & Weiss, 2005; Wandersman et al., 
2008). Similarly, organizational capacity is an 
area of growing interest within the field of 
implementation science (Fixsen, Naoom, 
Blase, Friedman, & Wallace, 2005; Glisson & 
Green, 2006). For example, it has been posited 
that schools with a high rate of disorder will 
struggle to implement programs with fidelity 
(Domitrovich et al., 2008). There is also some 
empirical literature that supports this hypoth¬ 
esis. Specifically, research has shown that 
schools’ structural composition (e.g., school 
size, ratio of students to teacher) and student 
demographic composition (e.g., high concen¬ 
tration of poverty and mobility)—which relate 
to negative student outcomes and serve as 
proxies for school disorder (e.g., Bradshaw, 
Sawyer, & O’Brennan, 2009; Gottfredson & 
Gottfredson, 2002)—hinder implementation 
(Beets et al., 2008; Bradshaw, Koth, et al., 
2009; Domitrovich et al., 2008; Hoy & Feld¬ 
man, 1987; Payne, Gottfredson, & Gottfred¬ 
son, 2006). Greater student mobility has also 
been associated with lower implementation fi¬ 
delity of Tier 1 PBIS in elementary and middle 
schools (Pas & Bradshaw, 2012). 

In contrast, elementary and middle 
schools with more qualified (i.e., certified) 
teachers achieved better implementation of 
PBIS (Pas & Bradshaw, 2012). Similarly, 
mental health services research has shown that 
environments that are more efficient, have 
positive norms, and have shared beliefs about 
implementation support the delivery of higher 
quality services (Glisson & Green, 2006; 
Schoenwald & Hoagwood, 2001). Taken to¬ 
gether, these findings regarding the link be¬ 
tween school context and implementation fi¬ 
delity suggest that it would be important to 
consider baseline school contextual factors 
when examining the adoption and rollout of 
PBIS, particularly in high schools, which gen¬ 
erally are larger, are more diverse, and have 
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higher levels of disorder than elementary 
schools (Bradshaw, Sawyer, et al., 2009; 
Domitrovich et al., 2008; Molloy, Moore, 
Trail, Van Epps, & Hopfer, 2013). 

CURRENT STUDY 

The PBIS model has been widely dis¬ 
seminated throughout the United States, with 
an estimated 22,000 schools currently imple¬ 
menting the Tier 1 supports. Moreover, much 
of the focus of PBIS implementation and out- 
comes-based research has been on elementary 
schools. As a result, less is known about the 
implementation of PBIS and the integration of 
other EBPs across the tiers, especially in high 
schools. The current study aimed to expand 
our understanding of the implementation of 
PBIS in high schools. Specifically, we exam¬ 
ined how the fidelity with which high schools 
implement PBIS changed over time and ex¬ 
plored whether school-level indicators of 
school disorder, including bullying, predicted 
implementation of PBIS. 

We used data from 31 high schools im¬ 
plementing PBIS that were part of a larger 58 
high school randomized trial of PBIS. This 
model of PBIS combined training across all 
three tiers, data-based decision-making using 
school climate and bullying data, and coach¬ 
ing supports. Given our interest in understand¬ 
ing the adoption and implementation of PBIS, 
we focused on the 31 schools randomized to 
the intervention condition. Specifically, our 
first aim described the rollout of PBIS by 
reporting school-level data assessing the core 
features of PBIS implementation across 2 
years, using two implementation fidelity tools. 
We then leveraged baseline data on bullying 
and other school-level indicators of school 
disorder (e.g., suspensions, mobility rates, stu¬ 
dent-teacher ratio, certified teachers) to ex¬ 
plore the extent to which these factors pre¬ 
dicted implementation of PBIS across the 
first 2 years of the trial. On the basis of prior 
efficacy studies of PBIS, we hypothesized that 
there would be some baseline features of PBIS 
in place prior to formal training in PBIS but 
that implementation of the core features would 
increase significantly after formal training and 


as schools gain more experience implementing 
the model (Bradshaw & Pas, 2011; Bradshaw, 
Reinke, Brown, Bevans, & Leaf, 2008; Pas & 
Bradshaw, 2012). We also hypothesized that 
schools with greater baseline disorder (e.g., 
students per teacher, higher mobility rates, and 
greater prevalence of bullying and suspen¬ 
sions) would have greater difficulty imple¬ 
menting PBIS (Bradshaw & Pas, 2011; Pas & 
Bradshaw, 2012). 

This study is novel, in part because of 
the setting in which it is conducted. Much of 
the extant randomized research on PBIS has 
been conducted in elementary school settings; 
to our knowledge, no rigorous randomized 
trials of PBIS have been conducted in high 
schools. The design of the trial allowed for 
longitudinal exploration of implementation of 
PBIS. This, in turn, may provide insight into 
potential areas of quick adoption of PBIS fea¬ 
tures and tiers, as well as potential challenges 
in implementation over time. We also ex¬ 
plored the extent to which contextual factors 
previously linked with implementation at the 
elementary school level (Domitrovich et al., 
2008) were also associated with implementa¬ 
tion in high schools. Finally, we focused on 
bullying as a particular form of school disor¬ 
der in this study, given its link with school 
climate and findings from a prior randomized 
trial in elementary schools (Waasdorp et al., 
2012) suggesting that it is a potentially impor¬ 
tant factor to be considered in relation to 
PBIS. 

METHOD 

Data from this study came from 31 tra¬ 
ditional (i.e., Grades 9-12) high schools 
across 12 Maryland counties, which were in¬ 
volved as intervention schools in a group ran¬ 
domized controlled effectiveness trial (Mur¬ 
ray, 1998) testing PBIS as part of the Mary¬ 
land Safe and Supportive Schools (MDS3) 
initiative. The 27 control (i.e., comparison or 
business as usual) schools were not included 
in the current study because they did not re¬ 
ceive training in PBIS. The 31 schools had a 
diverse student population, with a racial mi¬ 
nority rate of 45.7% and a mean student en- 
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Table 1. MDS3 School-Level 
Demographics for the 31 
Intervention Schools 


School Characteristics 

M 

SD 

School enrollment 

1,331.6 

488.9 

School attendance (%) 

92.8 

2.0 

Student mobility (%) 

18.6 

10.9 

Free or reduced-price meals (%) 

35.3 

16.4 

Special education (%) 

5.1 

0.3 

Racial minority (%) 

45.7 

25.7 

Student-teacher ratio 

19.8 

3.0 

Suspension rate (%) 

28.6 

17.5 

Algebra passing rate on HSA (%) 

87.8 

8.1 

Biology passing rate on HSA (%) 

85.6 

8.1 

English passing rate on HSA (%) 

85.5 

7.5 


Note. HSA = High School Assessment (state’s standard¬ 
ized academic test); MDS3 = Maryland Safe and Sup¬ 
portive Schools. 


rollment of 1,331.6 (SD = 488.9; see Table 1 
for sample demographic data). Approxi¬ 
mately 35.3% of the students at the participat¬ 
ing schools received free or reduced-priced 
lunch, and 5.1% received special education 
services. 

Study Design 

The project was led by the Maryland 
State Department of Education (MSDE) and 
included two subcontracts: one to a private, 
nonprofit mental health provider for training 
and implementation supports (i.e., imple¬ 
mentation partner) and the other to a team of 
university-based researchers for research 
design and evaluation supports (i.e., re¬ 
search partner). The MSDE led recruitment 
for the project; they approached districts 
based on need and willingness to participate. 
All approached districts consented to partic¬ 
ipate. Following district recruitment, princi¬ 
pals were approached about enrolling their 
schools and were asked to provide written 
consent for participation. The 58 recruited 
schools were randomized, by use of a group 
randomized controlled design (Murray, 
1998), following the baseline (spring) ad¬ 


ministration of the MDS3 School Climate 
Survey and fidelity assessments. A slightly 
higher proportion of schools (n = 31) was 
randomly assigned to the intervention con¬ 
dition to afford more statistical power when 
examining the implementation-related re¬ 
search questions of this study. Baseline data 
and 2 years of postintervention training data 
were analyzed in the current study. 

The 31 schools randomized to the inter¬ 
vention condition received training in the 
PBIS model as a framework for implementing 
an MTSS by the state’s implementation part¬ 
ner. Following the initial training in the three¬ 
tiered PBIS model (Sugai & Horner, 2006), 
intervention schools used the baseline school 
climate data to select from a menu of EBPs 
(see “Training and Support to Intervention 
Schools” subsection and Table 2). The MDS3 
initiative provided coaching supports and the 
necessary resources to implement one or more 
of the EBPs in the intervention schools. The 
researchers’ institutional review board ap¬ 
proved this study. For additional details on the 
trial design, see Bradshaw, Debnam, et al. 
(2014). 

Data Collection 

Students anonymously provided infor¬ 
mation about their experiences with bullying 
in the school via a password-protected, web- 
based survey (see details on the MDS3 School 
Climate Survey later). Each spring, a random 
sample of 25 classrooms of students (Grades 
9-12) was selected for participation in each 
school (i.e., seven 9th-grade language arts 
classrooms and six language arts classrooms 
per grade for 10th- to 12th-grade language arts 
classrooms). Student participation was ob¬ 
tained through a waiver of active parental con¬ 
sent and youth assent process. Site visits were 
conducted annually by contractors hired, 
trained, and supervised by the research partner 
to assess the fidelity with which PBIS was 
implemented across all project schools; all site 
visitors were unaware of the school’s interven¬ 
tion status. 
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Table 2. Summary of Evidence-Based Programs Implemented Through 

MDS3 Initiative 


Evidence-Based Program Tier 


Brief Overview of Program or Model 


Team-based process 

Emphasis on collection of data (e.g., bullying surveys) to 
inform program implementation 
Uses lessons, classroom meetings, and school-wide 
expectations regarding bullying 
Aims to change social norms regarding bullying 
Includes parent and community partners in prevention 
process 

Classroom-based social skills training program 
Primarily teacher facilitated 
Provides instruction, demonstration, feedback, 
reinforcement, and practice 

Covers personal self-management skills, general social 
skills, and drug resistance skills 
10 class sessions, approximately 40-45 min each 
Check-In/Check-Out 2 Targeted intervention for students needing positive adult 

attention and reinforcement 
Builds on PBIS model to develop behavior plan for 
meeting school-wide behavioral expectations 
Home/school communication 
Facilitated by trained staff (e.g., paraprofessionals) 

Check & Connect 2 Student engagement model 

School-based mentoring model to promote student 
engagement and attendance and prevent dropout 
Includes home visits and family-focused activities 
Facilitated by trained staff (e.g., paraprofessionals, pupil 
personnel workers) 

Cognitive-Behavioral 3 Focused on youth at greatest risk for behavioral and 

Intervention for Trauma substance abuse problems 

in Schools Promotes use of effective coping strategies 

Facilitated by trained student services staff (e.g., 
counselors, social workers, school psychologists) 

10 sessions focused on education about reactions to 
trauma, relaxation training, stress or trauma exposure, 
and social problem-solving 

Family and community focused and culturally sensitive 
and relevant for urban, racial minority youth 


Olweus Bullying Prevention 1 

Program 


Botvin LifeSkills Training 1 

(High School Program) 


Note. MDS3 = Maryland Safe and Supportive Schools; PBIS = Positive Behavioral Interventions and Supports. 


Training and Support to Intervention 
Schools 

Consistent with the PBIS framework, 
a train-the-trainer framework was used, 
whereby the PBIS teams received initial train¬ 


ing and implementation materials from 
coaches and national trainers, whereas the 
training activities at the school level were led 
by the school-specific PBIS teams with sup¬ 
port from project-assigned coaches. Specifi¬ 
cally, each school was assigned an interven- 
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tion support coach, who served three schools. 
The coaches were expected to spend approxi¬ 
mately two days per week working with each 
of their three assigned schools to support high- 
quality implementation of the prevention pro¬ 
grams. For the initial training for the 31 MDS3 
intervention schools, 10 personnel (i.e., school 
psychologists, teachers, administrators) from 
each school were brought together to serve as 
the MDS3 school-based leadership team. This 
first training included a large-group overview 
of the content of the initiative, an explanation 
of the targeted domains of school climate (i.e., 
safety, engagement, environment), and infor¬ 
mation on how to access and use the online 
MDS3 School Climate Survey System to fa¬ 
cilitate data-based decision-making. The train¬ 
ing also provided detailed information about 
each EBP being offered (discussed in the next 
paragraph) and included activities, opportuni¬ 
ties for practice, and action planning. 

Following the initial large-scale train¬ 
ing, coaches worked with their three imple¬ 
menting high schools over the course of the 
project. The coaches co-led trainings at the 
schools with school-level PBIS team mem¬ 
bers; the coaches also provided on-site tech¬ 
nical assistance relevant to high schools and 
the specific schools they served to promote 
high-fidelity implementation of PBIS. The 
coaching model was collaborative and focused 
on supporting the school teams. Using data 
from the web-based MDS3 School Climate 
Survey, which was specifically developed for 
use in high schools, coaches assisted school- 
level PBIS teams in the selection of the EBPs 
to implement within the PBIS framework 
(Bradshaw, Bottiani, et al., 2014). Each coach 
was trained in the relevant EBPs, consistent 
with the program developers’ specifications. 
The menu of EBPs included the Olweus Bul¬ 
lying Prevention Program (Olweus et al., 
2007), which is a Tier 1 approach for prevent¬ 
ing bullying; the LifeSkills Training for High 
Schools Program (Botvin, Griffin, & Nichols, 
2006), which is a Tier 1 substance abuse pre¬ 
vention model; Check-In/Check-Out (Hawken 
& Horner, 2003), which is a Tier 2 model for 
preventing behavior problems within the PBIS 
framework; Check & Connect (Sinclair, 


Christenson, & Thurlow, 2005), which is a 
Tier 2 mentoring model that emphasizes the 
school-home connection through trained 
school-based mentors; and Cognitive-Behav¬ 
ioral Intervention for Trauma in Schools 
(Stein et al., 2003), which provides a Tier 3, 
group-based preventive intervention to youth 
exposed to trauma and who are at risk of 
developing a behavioral or mental health prob¬ 
lem. See Table 2 for details. 

A subset of members of the MDS3 
school-based team from all intervention 
schools attended a centralized, annual 2-day 
summer booster training session. The majority 
of the EBP trainings were delivered regionally 
or on site, specifically targeting each school’s 
needs. School-based personnel were trained to 
implement PBIS and the EBPs by the coach in 
conjunction with the PBIS team; this train-the- 
trainer implementation framework is used to 
bolster internal implementation capacity and 
increase sustainability of the EBPs. Finally, 
coaching and technical assistance were pro¬ 
vided to the school district to encourage high- 
quality implementation and sustainability of 
the EBPs. 

Fidelity Measures 

Two fidelity measures served as the out¬ 
comes of interest for the current study. The 
School-wide Evaluation Tool (SET; Sugai, 
Lewis-Palmer, Todd, & Horner, 2001) is an 
observational tool used to assess the degree to 
which schools implement the key Tier 1 fea¬ 
tures of PBIS (Horner et al., 2004). The SET 
involves an observer conducting brief inter¬ 
views with staff and students, touring the 
school, and reviewing materials to document 
evidence of implementation. The external ob¬ 
server incorporated this information and pro¬ 
vided ratings for a series of items that com¬ 
prise seven sub scales on a scale of 0 (not 
implemented) to 2 {fully implemented). The 
scales are as follows: (a) expectations defined, 
(b) behavioral expectations taught, (c) system 
for rewarding behavioral expectations, (d) sys¬ 
tem for responding to behavioral violations, 
(e) monitoring and decision making, (f) man¬ 
agement, and (g) district-level support (see 
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Horner et al., 2004). Scores on the sub scales 
were calculated by summing all points earned, 
dividing by the total possible points, and mul¬ 
tiplying by 100. Therefore, scores range from 
0% to 100%, with higher scores indicating 
greater fidelity. An overall summary score was 
computed by averaging all seven scores (re¬ 
ferred to as the overall SET score), which also 
ranges from 0% to 100% (Cronbach’s 
a = 0.93 in the current study). An overall SET 
score of 80% or higher is considered high 
fidelity (Horner et al., 2004; Sugai et al., 
2001). The SET is the most widely used re- 
search-based measure of PBIS Tier 1 imple¬ 
mentation; it has been previously validated in 
several studies of PBIS (Bradshaw, Reinke, et 
al., 2008; Horner et al., 2004; Pas & Brad¬ 
shaw, 2012). The current trial used identical 
training and reliability assessment procedures 
to those used by Debnam et al. (2012). Spe¬ 
cifically, all trained observers participated in a 
2-day SET training and reached and main¬ 
tained at least 80% reliability with a trainer to 
participate in data collection for the project. 
All observers were unaware of the school’s 
intervention status (for additional details on 
the SET, training, and psychometric proce¬ 
dures, see Bradshaw, Reinke, et al., 2008; 
Debnam et al., 2012). 

The Individual Student Systems Evalu¬ 
ation Tool (I-SSET, Version 1.2; Lewis- 
Palmer, Todd, Horner, Sugai, & Sampson, 
2005) was similarly administered by the ex¬ 
ternal assessor to document the support ser¬ 
vices provided at the more advanced tiers (i.e., 
Tiers 2 and 3). The I-SSET includes four sub¬ 
scales: (a) the foundations subscale measured 
the basic processes and procedures in place for 
individual student systems; (b) the school¬ 
wide interventions subscale included specific 
questions about school-wide social-emotional 
and behavioral interventions in place at the 
school; (c) the targeted interventions subscale 
included specific questions about Tier 2 and 3 
behavioral interventions in the school; and (d) 
the intensive individualized interventions sub¬ 
scale measured the quality of the school’s 
team members involved in functional behav¬ 
ioral assessments. The I-SSET is similar in 
structure, form, and administration to the SET. 


For example, as with the SET, the observer 
provided ratings on a scale of 0 to 2 for each 
item and the percentage of components imple¬ 
mented was calculated for each scale. An 
overall I-SSET score was created by averaging 
the four subscale scores (0%-100%); again, 
higher scores indicate better implementation 
of the MTSS (a = 0.92 in the current study). 
Research by Debnam et al. (2012) indicated 
that the I-SSET is a valid and reliable measure 
of the fidelity of advanced-tier EBPs. All 
trained observers participated in a 2-day I- 
SSET training and reached and maintained at 
least 80% reliability with a trainer to partici¬ 
pate in data collection for the project. All 
observers were unaware of the school’s inter¬ 
vention status; for additional details on the 
I-SSET, training, and psychometrics, see Deb¬ 
nam et al. (2012). 

School-Level Indicators of Disorder 

Information regarding school disorder 
(i.e., data about students as well as staff; use of 
exclusionary discipline practices) were col¬ 
lected from the MSDE. These included the 
percentage of suspension events (i.e., total 
number of suspensions divided by student en¬ 
rollment), student mobility rate (i.e., total en¬ 
trants and withdrawals divided by student en¬ 
rollment), student-to-teacher ratio, and per¬ 
centage of teachers in the school with a 
standard or advanced teaching certification. 
These school-level indicators of school disor¬ 
der have been used in prior research examin¬ 
ing the association between school character¬ 
istics, PBIS adoption, and implementation at 
the elementary and middle school levels (e.g., 
Bradshaw & Pas, 2011; Pas & Bradshaw, 
2012). Another source of school disorder data 
came from the MDS3 School Climate Survey 
(Bradshaw, Waasdorp, Debnam, & Lindstrom 
Johnson, 2014), through which students re¬ 
ported their experience of bullying victimiza¬ 
tion within the past 30 days, including five 
forms of verbal victimization (e.g., being 
called names; being threatened; being teased) 
and three forms of physical victimization (i.e., 
being pushed or shoved; being hit, slapped, or 
kicked). These indicators were adapted from 
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previously published and validated measures 
of bullying (see Bradshaw, Sawyer, et al., 
2009; Sawyer, Bradshaw, & O’Brennan, 
2008; Solberg & Olweus, 2003). 

Overview of Analyses 

A series of descriptives, repeated-mea¬ 
sures analyses of variance (ANOVAs), and 
correlational analyses were conducted in 
SPSS, Version 22.0, to first examine trends in 
the implementation of the MTSS across three 
school years (i.e., baseline and the first two 
intervention years of the study), using all sub¬ 
scales of the implementation measures (i.e., 
SET and I-SSET). Specifically, a single re¬ 
peated-measures multivariate analysis of vari¬ 
ance (MANOVA) was run for the SET and 
I-SSET subscales to determine whether there 
was an overall change across time on the 
measures and whether there was a scale-by- 
time interaction. Post hoc repeated-measures 
ANOVAs were then used to examine the un¬ 
adjusted change for all SET and I-SSET sub¬ 
scales individually; to adjust for the number of 
tests conducted (i.e., MANOVAs and post hoc 
ANOVAs), we applied a conservative Bonfer- 
roni correction of p = .004. We then examined 
the zero-order correlations between the base¬ 
line rates of bullying and other school-level 
indicators of disorder with scores on the over¬ 
all SET and I-SSET scales for each time point 
and for change scores. 

Hierarchical linear modeling (HLM; 
Raudenbush & Bryk, 2002) was used to ex¬ 
amine the association between the repeated 
measures of the overall SET and I-SSET 
scores (i.e., at baseline, followed by Year 1 
after training and Year 2 after training) and 
baseline bullying prevalence and other indica¬ 
tors of school-level disorder. The repeated 
measures of the SET and I-SSET scores were 
modeled in separate models at Level 1. First, 
models were run with no predictors; then, just 
the uncentered time variable was added to 
Level 1 (i.e., 0 for baseline, 1 for Year 1 of 
intervention, and 2 for Year 2). At Level 2, the 
error term for the slope of time was freed to 
allow for the modeling of random effects of 
time between schools. The statistical tests in¬ 


dicated that the random variance for time was 
nonsignificant, and thus, it was fixed in sub¬ 
sequent models. The school-level variables 
were modeled at Level 2 and included the 
percentage of students who reported that they 
were verbally bullied, the percentage of stu¬ 
dents who reported that they were physically 
bullied, the suspension rate, the student- 
teacher ratio, the student mobility rate, and the 
percentage of teachers with certification. All 
Level 2 variables were grand mean centered 
(Luke, 2004). Two separate models were an¬ 
alyzed; the first model included bullying and 
other school-level disorder characteristics pre¬ 
dicting the intercept to examine the associa¬ 
tion of preintervention level of implementa¬ 
tion of PBIS, and the second model examined 
whether bullying and other school-level char¬ 
acteristics measured at baseline related to the 
change in implementation over time by mod¬ 
eling all predictors on the slope of time. These 
HLM analyses allowed us to determine 
whether baseline rates of bullying and school 
disorder were associated with implementation, 
as indicated by overall SET and I-SSET 
scores. Changes in the intraclass correlation 
coefficients (ICCs) and deviance were used to 
examine the variability explained by the 
models. 

RESULTS 

Statistics of central tendency were com¬ 
puted to assess the level of implementation as 
indicated by both the SET and I-SSET at each 
time point. The means for the SET and I-SSET 
subscale scores at baseline and at the end of 
Year 1 and Year 2 are presented in Table 3. 

SET and I-SSET Descriptive Information 

The average implementation score on 
the SET was below the 80% threshold at base¬ 
line and after 1 year of implementation for all 
but two subscales (i.e., system for responding 
to behavioral violations, as well as monitoring 
and decision making; for overall SET scores 
at baseline and Year 1, M = 60.59 and 
M = 70.63, respectively). After 2 years of 
implementation two subscales (i.e., behavioral 
expectations taught and system for rewarding 
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Table 3. Repeated-Measures ANOVAs for SET and I-SSET Scale Scores 


Within-Subject 

Baseline Year 1 Year 2 Effects 



M 

SD 

M 

SD 

M 

SD 

F 

P 

SET scales 

Expectations defined 

46.77 b ’° 

41.70 

72.58“ 

37.28 

82.26“ 

27.53 

11.13 

< .01* 

Expectations taught 

47.74 b ’° 

35.19 

62.90“-° 

31.33 

78.06“- b 

24.42 

14.06 

< .01* 

Rewarding expectations 

33.87° 

33.19 

46.24° 

35.15 

61.83“’ b 

35.80 

8.17 

< .01* 

Responding to violations 

81.61 b -° 

16.55 

90.32” 

11.69 

93.23“ 

7.91 

7.33 

< .01* 

Monitoring 

83.47 

20.00 

82.66° 

23.20 

91.94 b 

14.63 

3.59 

.03 

Management 

69.35° 

32.13 

70.36° 

33.46 

86.29“- b 

18.64 

4.20 

.02 

District-level support 

61.29° 

42.25 

69.35 

40.16 

83.87“ 

29.96 

3.47 

.04 

Overall SET score 

60.59 b -° 

23.70 

70.63“-° 

23.24 

82.50“- b 

16.21 

15.51 

< .01* 

I-SSET scales 

Foundations 

68.50 

15.47 

71.11 

11.94 

76.25 

18.06 

2.04 

.14 

School-wide interventions 

58.06 

29.02 

55.88 

31.24 

62.79 

27.29 

0.67 

.51 

Targeted interventions 

37.47 b, ° 

31.75 

61.29“-° 

32.21 

79.53“ ,b 

25.52 

18.31 

< .01* 

Intensive interventions 

76.81 b -° 

24.00 

86.49“ 

14.97 

91.94“ 

9.44 

7.50 

< .01* 

Overall I-SSET score 

60.24° 

19.52 

68.69° 

16.94 

77.62“' b 

14.33 

10.36 

< .01* 


Note. ANOVA = analysis of variance; I-SSET = Individual Student Systems Evaluation Tool; SET = School-wide 
Evaluation Tool. 

a Significant pairwise comparison versus baseline. 
b Significant pairwise comparison versus Year 1. 
c Significant pairwise comparison versus Year 2. 

* Significant at p < .004 level (i.e., Bonferroni correction). 


behavioral expectations) remained below the 
80% benchmark, whereas all other scales were 
above this 80% benchmark (for overall SET 
score, M = 82.50; see Table 3 for listing of all 
subscale score means). By Year 2, 71% of the 
schools reached the benchmark of achieving 
80% or higher on the overall SET score. On 
the I-SSET, only the average score on the 
intensive individualized interventions subscale 
reached 80% or higher by the end of the first 
year. After 2 years, none of the other scales 
had reached this benchmark, on average, al¬ 
though the mean scores for the targeted inter¬ 
ventions subscale and overall I-SSET were 
similar, at 79.53 and 77.62, respectively. After 
the second year of implementation, only 45% 
of the schools had reached the 80% fidelity 
benchmark on the overall 1-SSET scale. 

The repeated-measures MANOVA indi¬ 
cated a significant effect for time, F( 2, 
56) = 27.03, p < .001, r\ 2 = 0.49; subscale, 
E(10, 48) = 40.39, p < .001, -q 2 = 0.89; 
and time-by-subscale interaction, F(20, 


38) = 6.34, p < .001, T) 2 = 0.78. It is worth 
highlighting that the effect size is rather siz¬ 
able, particularly in light of the relatively 
small sample size. On the basis of the overall 
significant MANOVA results, we then con¬ 
ducted post hoc repeated-measures ANOVAs 
to compare the individual SET and I-SSET 
subscale scores across time; this enabled us to 
determine specifically which scales changed 
significantly over time. The repeated-mea¬ 
sures ANOVAs showed that most SET sub¬ 
scale scores improved significantly over time. 
In particular, there were immediate and con¬ 
tinued improvements on the subscale for be¬ 
havioral expectations taught (i.e., all pairwise 
comparisons reflected significant increases), 
as well as the overall SET score. Specifically, 
the scores improved after both the first and 
second years of implementation. Significant 
differences emerged after the first year of im¬ 
plementation for the subscale for expectations 
defined and the subscale for system for re¬ 
sponding to behavioral violations (i.e., differ- 
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Table 4. Correlations Between the SET and I-SSET Implementation 
Measures for the 31 Intervention Schools 



1 

2 

3 

4 

5 

6 

7 

8 

9 

1. Baseline SET 










2. Baseline I-SSET 

.325 









3. Year 1 SET 

.556** 

-.059 








4. Year 1 I-SSET 

.240 

.133 

.555** 







5. Year 2 SET 

.383* 

.056 

.504** 

.420* 






6 . Year 2 I-SSET 

.398* 

.271 

.357* 

.308 

.345 





7. SET 1-year change 

-.487** 

-.410* 

.455* 

.326 

.119 

-.052 




8 . SET 2-year change 

-.760** 

-.295 

-.218 

.048 

.309 

-.167 

.585** 



9. I-SSET 1-year change 

-.095 

_ 717 ** 

.438* 

.596** 

.250 

-.003 

.562** 

.273 


10. I-SSET 2-year change 

-.031 

-.750** 

.300 

.087 

.184 

.433* 

.349 

.162 

.669** 


Note. I-SSET = Individual Student Systems Evaluation Tool; SET = School-wide Evaluation Tool. 
*p < .05. **p < .01. 


ences between baseline and Year 1); however, 
no further significant improvements emerged 
when we compared the second year of imple¬ 
mentation with the first. As noted earlier, on 
average, schools reached the 80% benchmark 
after the first year on the subscale for system 
for responding to behavioral violations; this 
may explain why no further improvements on 
this subscale occurred after the first year. The 
subscale for rewarding behavioral expecta¬ 
tions only improved significantly after 2 years 
of implementation. The remaining three scales 
(i.e., monitoring and decision making, man¬ 
agement, and district support) did not change 
significantly when the Bonferroni-corrected p 
value was used. 

Statistically significant improvements 
on I-SSET subscale scores were demonstrated 
on the targeted interventions subscale and the 
intensive individualized interventions sub¬ 
scale, as well as overall scale scores (i.e., but 
not on the foundations subscale or school¬ 
wide interventions subscale). Pairwise com¬ 
parisons indicated that significant improve¬ 
ments emerged on the targeted interventions 
subscale, as well as the intensive individual¬ 
ized interventions scale (i.e., in comparing 
baseline with Year 1), whereas the overall 
score on the scale improved later (i.e., from 
Year 1 to Year 2). Scores on the targeted 
interventions subscale also improved further 
between Year 1 and Year 2. 


For efficiency, we focused the remain¬ 
ing statistical tests only on overall SET and 
I-SSET scores, which reflect Tier 1 and com¬ 
bined Tier 1, 2, and 3 scores, respectively. 
Although the averages on the SET and I-SSET 
were comparable and not statistically signifi¬ 
cantly different when contrasted to one an¬ 
other across time points, they were only sig¬ 
nificantly correlated with one another at the 
end of the first year of the trial (r = .56, p < 
.01). The change scores for the first year were 
also significantly correlated with one another 
(r = .56, p < .01). Correlations between the 
SET and I-SSET at baseline and after Year 2 
approached significance at the p < .10 level. 
See Table 4 for additional details regarding the 
correlational findings. 

Correlations Between Implementation 
and School Variables 

We also examined the correlations be¬ 
tween the overall SET and 1-SSET scores at 
baseline. Year 1, and Year 2, as well as the 
change scores from baseline to Year 1 and 
from baseline to Year 2, with bullying preva¬ 
lence indicators and other school-level char¬ 
acteristics (see Table 5). Baseline bullying in¬ 
dicators were significantly associated with the 
Year 1 SET and 1-SSET scores. Specifically, 
schools with a higher prevalence of student- 
reported verbal victimization (r = .50, p < 
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Table 5. Correlations Between Implementation and School Characteristics 
among the 31 Intervention Schools 


Baseline 

Characteristics 

Baseline 

SET 

Baseline 

I-SSET 

Year 1 
SET 

Year 1 
I-SSET 

Year 2 
SET 

Year 2 
I-SSET 

SET 

1-Year 

Change 

SET 

2 -Year 
Change 

I-SSET 

1-Year 

Change 

1- SSET 

2- Year 
Change 

Prevalence of verbal 

bullying 

.164 

-.056 

.495** 

.339 

.320 

.067 

.345 

.057 

.284 

.098 

Prevalence of being 

physical bullying 

.242 

.047 

464** 

.534** 

.495** 

.207 

.228 

.099 

.338 

.098 

Suspension rate 

.085 

-.142 

.102 

-.094 

.339 

.100 

.016 

.151 

.049 

.202 

Student-teacher ratio 

.002 

.061 

.265 

.242 

.053 

.019 

.276 

.035 

.121 

-.044 

Percentage of certified 

teachers 

-.059 

-.028 

-.097 

-.144 

-.372* 

-.299 

-.039 

-.201 

-.078 

-.179 

Student mobility 

.128 

.164 

-.083 

-.262 

.233 

.170 

-.224 

.033 

-.318 

-.037 


Note. I-SSET = Individual Student Systems Evaluation Tool; SET = School-wide Evaluation Tool. 
*p < .05. **p < .01. 


.05) and physical victimization (r = .46, p < 
.05) had significantly higher SET scores. 
Schools with a higher baseline prevalence of 
students reporting physical victimization had 
higher SET scores in Year 2 (r = .50, p < 
.05). A significant association was also ob¬ 
served between the baseline prevalence of 
physical victimization and I-SSET score in 
Year 1 (r = .53, p < .05). The only significant 
correlation between other baseline school- 
level indicators of disorder and implementa¬ 
tion was the baseline percentage of certified 
teachers and the Year 2 SET scores (r = -.37, 
p < .05). None of the other baseline school 
characteristics were significantly associated 
with change scores on the overall SET and 
I-SSET. 

Multilevel Analyses 

Four final HLMs were fit, including two 
models that incorporated all school-level char¬ 
acteristics on the intercept and on the slope of 
the time variable for each of the two outcomes 
(i.e., SET and I-SSET scores; see Table 6). 
The covariate-adjusted intercept estimates 
were generally consistent with the baseline 
means: 60.28 for the SET and 60.16 for the 
I-SSET. None of the variables were signifi¬ 
cantly associated with the SET intercept. The 
rate of physical victimization was associated 
with higher I-SSET intercepts when we ad¬ 


justed for other indicators of school disorder. 
In addition, the suspension rate was inversely 
associated with the intercept of the I-SSET; 
specifically, the higher the suspension rate, the 
lower the overall I-SSET score. 

Consistent with the ANOVA findings, 
schools significantly improved on both the 
SET and I-SSET measures over time. Specif¬ 
ically, SET scores improved by 10.95 percent¬ 
age points (on a 100-point scale) across the 
2 years, whereas I-SSET scores improved 
by 8.69 percentage points (also on a 100-point 
scale). Contrary to our hypothesis, schools 
with a higher baseline rate of verbal victim¬ 
ization experienced significantly greater in¬ 
creases in their SET scores over the three data 
points. The baseline percentage of certified 
teachers was inversely associated with growth 
in both SET and I-SSET scores over time. 
There were no other significant associations 
between the other indicators of school disorder 
and the changes in SET and I-SSET scores 
over the course of the three waves of data. 
Reductions in the ICCs and deviance indicated 
that the final models improved fit and ex¬ 
plained a sizable amount of between-school 
variability in the outcomes (see Table 6). 

DISCUSSION 

The purpose of this study was to exam¬ 
ine the implementation of an MTSS frame- 
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Table 6. 

HLM Results 






SET 



I-SSET 



Coefficient 

SE 

P 

Coefficient 

SE 

P 

HLM results for predicting SET and 

I-SSET intercept 

Intercept 

60.28 

3.97 

< .01 

60.16 

2.79 

< .01 

Physical bullying, yOl 

1.26 

0.92 

.19 

2.02 

0.68 

.01 

Verbal bullying, y02 

1.40 

0.83 

.10 

-0.60 

0.58 

.31 

Suspension, y03 

0.05 

0.19 

.79 

-0.37 

0.15 

.02 

Student-teacher ratio, y04 

0.47 

0.98 

.64 

0.95 

0.55 

.10 

Teacher certification, y05 

-1.19 

0.90 

.20 

-1.26 

0.68 

.07 

Mobility, y06 

0.24 

0.41 

.56 

0.27 

0.25 

.28 

Time 

10.95 

2.03 

< .01 

8.69 

1.84 

< .01 

Variance indicators 

Reduction in ICC 

Initial deviance 

Final deviance 

18.28% 

802.07 

784.06 



38.96% 

802.07 

793.79 



HLM results for predicting SET and 

I-SSET slope 

Intercept 

60.28 

4.20 

< .001 

60.16 

3.05 

< .001 

Time 

10.95 

2.01 

< .001 

8.69 

1.85 

< .001 

Physical bullying, yll 

0.39 

0.33 

.23 

0.76 

0.48 

.12 

Verbal bullying, yl2 

0.70 

0.22 

< .01 

0.04 

0.36 

.91 

Suspension, yl3 

0.08 

0.08 

.36 

-0.08 

0.07 

.29 

Student-teacher ratio, yl4 

0.37 

0.45 

.41 

0.34 

0.39 

.39 

Teacher certification, 7 15 

-0.96 

0.29 

< .01 

-0.97 

0.27 

< .001 

Mobility, y !6 

-0.03 

0.13 

.81 

-0.05 

0.14 

.71 

Variance indicators 

Reduction in ICC 

Initial deviance 

Final deviance 

22.19% 

776.08 

762.71 



36.40% 

776.08 

771.82 




Note. Reductions in ICC reflect the proportion of change relative to initial variance components. HLM = hierarchical 
linear modeling; ICC = intraclass correlation coefficient; I-SSET = Individual Student Systems Evaluation Tool; 
SET = School-wide Evaluation Tool. 


work in high schools. The MTSS model tested 
in this study included training in the multit¬ 
iered PBIS model and access to data on school 
climate, which were used to guide the selec¬ 
tion and implementation of one or more EBPs, 
with ongoing support from a coach. As has 
been seen in studies of PBIS in elementary and 
middle schools (Bradshaw & Pas, 2011; Brad¬ 
shaw, Reinke, et ah, 2008; Pas & Bradshaw, 
2012), the current sample of high schools on 
average demonstrated increasing levels of im¬ 
plementation over the course of 2 years. Con¬ 
sistent with the MTSS framework, schools 
generally appeared to make greater gains in 
Tier 1 supports, as compared with advanced- 


tier supports. Specifically, SET scores 2 years 
after training demonstrated that 22 of the 31 
schools (71%) met or exceeded 80% fidelity 
on the Tier 1 supports and, on average, 
scored 82.50%. During this same period, 
fewer than half of the schools implemented 
80% of the advanced-tier components (as 
measured by the I-SSET), and the average 
I-SSET score was 77.62%. This rate of adop¬ 
tion of Tier 1 was roughly on par with Tier 1 
supports previously observed in PBIS studies 
at the elementary school level (Bradshaw, 
Koth, et al., 2009; Bradshaw, Reinke, et al., 
2008); however, the rate of adoption of ad¬ 
vanced-tier supports in this sample of high 


493 









School Psychology Review, 2015, Volume 44, No. 4 


schools appeared to be a bit slower than that in 
PBIS studies at the elementary school level 
(Bradshaw, Pas, et al., 2012). These findings 
highlight the need for high schools to set re¬ 
alistic expectations for the period required for 
program adoption that meets high-fidelity im¬ 
plementation, particularly regarding the time¬ 
line for advanced-tier implementation. 

Both the correlations and HLM analyses 
highlighted the importance of baseline preva¬ 
lence of bully victimization regarding imple¬ 
mentation of PBIS. Specifically, the preva¬ 
lence of victimization by bullying was signif¬ 
icantly related to Year 1 and Year 2 SET and 
I-SSET scores. The percentage of certified 
teachers was the only other variable that was 
significantly correlated with SET scores. 
These findings were replicated in the covari- 
ate-adjusted HLMs addressing our third re¬ 
search question regarding baseline factors that 
predict changes in fidelity of Tiers 1, 2, and 3 
over time. The adjusted multilevel results 
demonstrated a significant association be¬ 
tween the baseline rate of victimization by 
physical bullying and the intercept for the 
I-SSET. Additionally verbal bullying at base¬ 
line was related to the growth of SET scores 
over time. This finding was inconsistent with 
our hypothesis that greater disorder, as mea¬ 
sured in this case by bullying prevalence, 
would inhibit implementation. It is possible 
that the baseline prevalence of bullying may 
have served as a motivating factor for schools 
to implement more Tier 1 positive behavioral 
approaches. An interesting finding was that 
verbal victimization was associated with SET 
changes, whereas physical bullying rates were 
associated with I-SSET intercept scores only. 
In contrast, a higher rate of baseline suspen¬ 
sions was associated with lower fidelity scores 
on the I-SSET, reflecting less school focus on 
advanced-tier implementation relative to the 
universal, Tier 1 supports. Additional research 
is needed to replicate and further examine 
these trends. 

The baseline rate of certified teachers 
was inversely associated with Year 2 SET 
scores and with change in overall SET scores 
over time. This finding was unexpected be¬ 
cause prior research suggested a significantly 


positive association between teacher certifica¬ 
tion and implementation (Bradshaw & Pas, 
2011; Pas & Bradshaw, 2012). The difference 
in the school setting (high schools vs. elemen¬ 
tary and middle schools) may account for 
these unexpected findings. No other baseline 
predictors were significantly associated with 
changes in SET or I-SSET scores over time. 
Taken together, these findings suggest that 
baseline indicators of disorder were generally 
not associated with PBIS implementation and 
thus do not appear to be barriers to adoption. 

Limitations 

The sample size of 31 intervention 
schools may have limited the power to detect 
effects within the multilevel models. We fo¬ 
cused most of the statistical analyses on the 
continuous SET and I-SSET scores to better 
understand the overall pattern and growth in 
findings and leverage statistical power (Mur¬ 
ray, 1998); additional empirical work is 
needed to better understand the specific 
threshold at which PBIS implementation 
across all the three tiers translates into out¬ 
comes in high schools. Although there was 
randomization to intervention condition, as 
well as a set number of EBPs that schools had 
access to through the project, schools opted to 
implement the various EBPs at different rates. 
On average, the approach was to focus on 
Tier 1 in Year 1 and then layer on additional 
universal and advanced-tier EBPs in Year 2. 
The empirical findings regarding the rollout 
and implementation of the various tiers were 
generally consistent with recommended prac¬ 
tice (Barrett et al., 2008). The school-level 
PBIS team members, with support from their 
MDS3 coach, served as the school-level train¬ 
ers for the program elements. Although the 
coaches tracked their own coaching supports 
and time in the school, the amount of training 
time and supports provided to teachers and 
other school staff by the PBIS teams varied. 
Such variability in training and implementa¬ 
tion support is common in effectiveness stud¬ 
ies, such as this one (as compared to efficacy 
studies; Domitrovich et al., 2008; Gottfredson 
et al., 2015). 
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Nevertheless, there are a number of 
strengths of this study including our focus on 
high schools, the use of outside assessors to 
rate the fidelity of the PBIS elements, and the 
use of previously validated measures. Future 
research should further explore the psycho¬ 
metric properties of the SET and I-SSET at the 
high school level, given that the vast majority 
of the psychometric work on these instruments 
has been conducted at the elementary school 
level. For example, exploration of measure¬ 
ment invariance across grade levels may pro¬ 
vide further insight into the extent to which 
these tools are truly measuring the same core 
features across grade levels. Such an explora¬ 
tion is beyond the scope of the current study 
but is certainly an area of increasing interest 
within the burgeoning field of implementation 
science in relation to school-based prevention 
programming (e.g., Polanin & Espelage, 2014; 
Schulte, Easton, & Parker, 2009). It is impor¬ 
tant to note that the internal consistency esti¬ 
mates for the SET and 1-SSET were relatively 
high in this sample, suggesting that these tools 
are also appropriate for use in high schools. 

Conclusions and Implications 

This study is novel in its focus on PBIS 
in high schools, as well as the attention to 
PBIS adoption and implementation across 
all 3 tiers of the MTSS. Similarly, although 
PBIS implementation research has examined 
associations with school characteristics (e.g., 
Bradshaw & Pas, 2011; Pas & Bradshaw, 
2012), baseline rates of bullying have not been 
previously explored as a possible predictor of 
implementation. Specifically, by leveraging 
the MDS3 School Climate Survey, we ex¬ 
plored the extent to which victimization by 
different forms of bullying was associated 
with subsequent implementation. Our findings 
suggest that victimization stands out as a 
stronger predictor than other more readily 
available data (e.g., suspension rate); this sug¬ 
gests that baseline rates of bullying may be a 
motivator for, but not an impediment to, PBIS 
adoption and implementation. Furthermore, it 
may be advantageous for PBIS schools to 
make use of data systems that assess and track 


bullying throughout the implementation pro¬ 
cess (Bradshaw, 2013). Given their expertise 
in data-based decision-making and consulta¬ 
tion, school psychologists may play an impor¬ 
tant role in supporting schools’ use of various 
types of data, including information on school 
climate and bullying. These findings also 
highlight the growing interest in and attention 
to issues related to implementation within the 
field of school psychology (Forman et al., 
2013). Future studies based on the data from 
this trial will link these implementation find¬ 
ings with student outcomes and will explore 
changes in bullying and climate over time. 
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